Tutorial (week 2) B: text mining

Step 0: check and install needed packages. Load the libraries and functions.

library("sentimentr")

Warning message:
running command '"C:/Program Files/RStudio/bin/pandoc/pandoc" +RTS -K512m -RTS wk2-Tutorial-TextMining.utf8.md --to html --from markdown+autolink_bare_uris+ascii_identifiers+tex_math_single_backslash --output pandoc2b144ebbb5.html --smart --email-obfuscation none --self-contained --standalone --section-divs --template "C:\Users\JL\Documents\R\win-library\3.3\rmarkdown\rmd\h\default.html" --no-highlight --variable highlightjs=1 --variable "theme:bootstrap" --include-in-header "C:\Users\JL\AppData\Local\Temp\RtmpMFdQjt\rmarkdown-str2b141eae42de.html" --mathjax --variable "mathjax-url:https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML" --variable code_folding=show --variable source_embed=wk2-Tutorial-TextMining.Rmd --include-after-body "C:\Users\JL\AppData\Local\Temp\RtmpMFdQjt\file2b1456c31d17.html" --variable code_menu=1 --variable kable-scroll=1' had status 67 

This notebook was prepared with the following environmental settings.

print(R.version)

Step 1: Data harvest: scrap speech URLs from http://www.presidency.ucsb.edu/.

Following the example of Jerid Francom, we used Selectorgadget to choose the links we would like to scrap. For this project, we selected all inaugural addresses of past presidents, nomination speeches of major party candidates and farewell addresses. We also included several public speeches from Donald Trump for our textual analysis of presidential speeches.

### Inauguaral speeches
main.page <- read_html(x = "http://www.presidency.ucsb.edu/inaugurals.php")
# Get link URLs
# f.speechlinks is a function for extracting links from the list of speeches. 
inaug=f.speechlinks(main.page)
#head(inaug)
as.Date(inaug[,1], format="%B %e, %Y")

 [1] NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA
[30] NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA
[59] NA

inaug=inaug[-nrow(inaug),] # remove the last line, irrelevant due to error.
#### Nomination speeches
main.page=read_html("http://www.presidency.ucsb.edu/nomination.php")
# Get link URLs
nomin <- f.speechlinks(main.page)
#head(nomin)
#
#### Farewell speeches
main.page=read_html("http://www.presidency.ucsb.edu/farewell_addresses.php")
# Get link URLs
farewell <- f.speechlinks(main.page)
#head(farewell)

Step 2: Using speech metadata posted on http://www.presidency.ucsb.edu/, we prepared CSV data sets for the speeches we will scrap.

inaug.list=read.csv("../data/inauglist.csv", stringsAsFactors = FALSE)
nomin.list=read.csv("../data/nominlist.csv", stringsAsFactors = FALSE)
farewell.list=read.csv("../data/farewelllist.csv", stringsAsFactors = FALSE)

We assemble all scrapped speeches into one list. Note here that we don’t have the full text yet, only the links to full text transcripts.

Step 3: scrap the texts of speeches from the speech URLs.

speech.list=rbind(inaug.list, nomin.list, farewell.list)
speech.list$type=c(rep("inaug", nrow(inaug.list)),
                   rep("nomin", nrow(nomin.list)),
                   rep("farewell", nrow(farewell.list)))
speech.url=rbind(inaug, nomin, farewell)
speech.list=cbind(speech.list, speech.url)

Based on the list of speeches, we scrap the main text part of the transcript’s html page. For simple html pages of this kind, Selectorgadget is very convenient for identifying the html node that rvest can use to scrap its content. For reproducibility, we also save our scrapped speeches into our local folder as individual speech files.

# Loop over each row in speech.list
speech.list$fulltext=NA
for(i in seq(nrow(speech.list))) {
  text <- read_html(speech.list$urls[i]) %>% # load the page
    html_nodes(".displaytext") %>% # isloate the text
    html_text() # get the text
  speech.list$fulltext[i]=text
  # Create the file name
  filename <- paste0("../data/fulltext/", 
                     speech.list$type[i],
                     speech.list$File[i], "-", 
                     speech.list$Term[i], ".txt")
  sink(file = filename) %>% # open file to write 
  cat(text)  # write the file
  sink() # close the file
}

Trump, as president-elect that has not been a politician, do not have a lot of formal speeches yet. For our textual analysis, we manually add several public transcripts from Trump: + [Transcript: Donald Trump’s full immigration speech, annotated. LA Times, 08/31/2016] (http://www.latimes.com/politics/la-na-pol-donald-trump-immigration-speech-transcript-20160831-snap-htmlstory.html) + Transcript of Donald Trump鈥檚 speech on national security in Philadelphia - The Hill, 09/07/16 + Transcript of President-elect Trump’s news conference CNBC, 01/11/2017

Step 4: data Processing — generate list of sentences

We will use sentences as units of analysis for this project, as sentences are natural languge units for organizing thoughts and ideas. For each extracted sentence, we apply sentiment analysis using NRC sentiment lexion. “The NRC Emotion Lexicon is a list of English words and their associations with eight basic emotions (anger, fear, anticipation, trust, surprise, sadness, joy, and disgust) and two sentiments (negative and positive). The annotations were manually done by crowdsourcing.”

We assign an sequential id to each sentence in a speech (sent.id) and also calculated the number of words in each sentence as sentence length (word.count).

sentence.list=NULL
for(i in 1:nrow(speech.list)){
  sentences=sent_detect(speech.list$fulltext[i],
                        endmarks = c("?", ".", "!", "|",";"))
  if(length(sentences)>0){
    emotions=get_nrc_sentiment(sentences)
    word.count=word_count(sentences)
    # colnames(emotions)=paste0("emo.", colnames(emotions))
    # in case the word counts are zeros?
    emotions=diag(1/(word.count+0.01))%*%as.matrix(emotions)
    sentence.list=rbind(sentence.list, 
                        cbind(speech.list[i,-ncol(speech.list)],
                              sentences=as.character(sentences), 
                              word.count,
                              emotions,
                              sent.id=1:length(sentences)
                              )
    )
  }
}

Some non-sentences exist in raw data due to erroneous extra end-of sentence marks.

sentence.list=
  sentence.list%>%
  filter(!is.na(word.count)) 

Step 5: Data analysis — length of sentences

For simpler visualization, we chose a subset of better known presidents or presidential candidates on which to focus our analysis.

sel.comparison=c("DonaldJTrump","JohnMcCain", "GeorgeBush", "MittRomney", "GeorgeWBush",
                 "RonaldReagan","AlbertGore,Jr", "HillaryClinton","JohnFKerry", 
                 "WilliamJClinton","HarrySTruman", "BarackObama", "LyndonBJohnson",
                 "GeraldRFord", "JimmyCarter", "DwightDEisenhower", "FranklinDRoosevelt",
                 "HerbertHoover","JohnFKennedy","RichardNixon","WoodrowWilson", 
                 "AbrahamLincoln", "TheodoreRoosevelt", "JamesGarfield", 
                 "JohnQuincyAdams", "UlyssesSGrant", "ThomasJefferson",
                 "GeorgeWashington", "WilliamHowardTaft", "AndrewJackson",
                 "WilliamHenryHarrison", "JohnAdams")

Overview of sentence length distribution by different types of speeches.

Nomination speeches

First, we look at nomination acceptance speeches at major party’s national conventions. For relevant to Trump’s speeches, we limit our attention to speeches for the first terms of former U.S. presidents. We noticed that a number of presidents have very short sentences in their nomination acceptance speeches.

First term

par(mar=c(2, 5, 1, 1))
#sel.comparison=levels(sentence.list$FileOrdered)
sentence.list.sel=filter(sentence.list, 
                        type=="nomin", Term==1, File%in%sel.comparison)
sentence.list.sel$File=factor(sentence.list.sel$File)
sentence.list.sel$FileOrdered=reorder(sentence.list.sel$File, 
                                  sentence.list.sel$word.count, 
                                  mean, 
                                  order=T)
beeswarm(word.count~FileOrdered, 
         data=sentence.list.sel,
         horizontal = TRUE, 
         pch=16, col=alpha(brewer.pal(9, "Set1"), 0.6), 
         cex=0.55, cex.axis=0.8, cex.lab=0.8,
         spacing=5/nlevels(sentence.list.sel$FileOrdered),
         las=2, xlab="Number of words in a sentence.", ylab="",
         main="Nomination speeches")

Second term

par(mar=c(2, 5, 1, 1))
#sel.comparison=levels(sentence.list$FileOrdered)
sentence.list.sel=filter(sentence.list, 
                        type=="nomin", Term==2, File%in%sel.comparison)
sentence.list.sel$File=factor(sentence.list.sel$File)
sentence.list.sel$FileOrdered=reorder(sentence.list.sel$File, 
                                  sentence.list.sel$word.count, 
                                  mean, 
                                  order=T)
beeswarm(word.count~FileOrdered, 
         data=sentence.list.sel,
         horizontal = TRUE, 
         pch=16, col=alpha(brewer.pal(9, "Set1"), 0.6), 
         cex=0.55, cex.axis=0.8, cex.lab=0.8,
         spacing=1.2/nlevels(sentence.list.sel$FileOrdered),
         las=2, xlab="Number of words in a sentence.", ylab="",
         main="Nomination speeches, 2nd term")

What are these short sentences?

sentence.list%>%
  filter(File=="DonaldJTrump", 
         type=="nomin", 
         word.count<=3)%>%
  select(sentences)%>%sample_n(10)

sentence.list%>%
  filter(File=="AlbertGore,Jr", 
         type=="nomin", 
         word.count<=3)%>%
  select(sentences)%>%sample_n(10)

sentence.list%>%
  filter(File=="Clinton", 
         type=="nomin", 
         word.count<=3)%>%
  select(sentences)

sentence.list%>%
  filter(File=="WilliamJClinton", 
         type=="nomin", Term==1,
         word.count<=3)%>%
  select(sentences)

Inaugural speeches

We notice that the sentences in inaugural speeches are longer than those in nomination acceptance speeches.

sentence.list.sel=sentence.list%>%filter(type=="inaug", File%in%sel.comparison, Term==1)
sentence.list.sel$File=factor(sentence.list.sel$File)
sentence.list.sel$FileOrdered=reorder(sentence.list.sel$File, 
                                  sentence.list.sel$word.count, 
                                  mean, 
                                  order=T)
par(mar=c(2, 5, 1, 1))
beeswarm(word.count~FileOrdered, 
         data=sentence.list.sel,
         horizontal = TRUE,
         pch=16, col=alpha(brewer.pal(9, "Set1"), 0.6), 
         cex=0.55, cex.axis=0.8, cex.lab=0.8,
         spacing=5/nlevels(sentence.list.sel$FileOrdered),
         las=2, ylab="", xlab="Number of words in a sentence.",
         main="Inaugural Speeches")

Short sentences in inaugural speeches.

sentence.list%>%
  filter(File=="BarackObama", 
         type=="inaug", 
         word.count<=3)%>%
  select(sentences)

Step 5: Data analysis — sentiment analsis

Sentence length variation over the course of the speech, with emotions.

How our presidents (or candidates) alternate between long and short sentences and how they shift between different sentiments in their speeches. It is interesting to note that some presidential candidates’ speech are more colorful than others. Here we used the same color theme as in the movie “Inside Out.”

image

par(mfrow=c(4,1), mar=c(0,0,2,0), bty="n", xaxt="n", yaxt="n", font.main=1)
f.plotsent.len(In.list=sentence.list, InFile="HillaryClinton", 
               InType="nomin", InTerm=1, President="Hillary Clinton")
f.plotsent.len(In.list=sentence.list, InFile="DonaldJTrump", 
               InType="nomin", InTerm=1, President="Donald Trump")

f.plotsent.len(In.list=sentence.list, InFile="BarackObama", 
               InType="nomin", InTerm=1, President="Barack Obama")
f.plotsent.len(In.list=sentence.list, InFile="GeorgeWBush", 
               InType="nomin", InTerm=1, President="George W. Bush")

What are the emotionally charged sentences?

print("Hillary Clinton")

[1] "Hillary Clinton"

speech.df=tbl_df(sentence.list)%>%
  filter(File=="HillaryClinton", type=="nomin", word.count>=4)%>%
  select(sentences, anger:trust)
speech.df=as.data.frame(speech.df)
as.character(speech.df$sentences[apply(speech.df[,-1], 2, which.max)])

[1] "Some of you are frustrated, even furious."                        
[2] "It's a big deal."                                                 
[3] "Powerful forces are threatening to pull us apart."                
[4] "Powerful forces are threatening to pull us apart."                
[5] "It's a big deal."                                                 
[6] "My mother, Dorothy, was abandoned by her parents as a young girl."
[7] "It's a big deal."                                                 
[8] "Bonds of trust and respect are fraying."                          

print("Barack Obama")

[1] "Barack Obama"

speech.df=tbl_df(sentence.list)%>%
  filter(File=="BarackObama", type=="nomin", Term==1, word.count>=5)%>%
  select(sentences, anger:trust)
speech.df=as.data.frame(speech.df)
as.character(speech.df$sentences[apply(speech.df[,-1], 2, which.max)])

[1] "They could've heard words of anger and discord."
[2] "And that's to be expected."                     
[3] "It's not because John McCain doesn't care."     
[4] "Now let there be no doubt."                     
[5] "That promise is our greatest inheritance."      
[6] "Now let there be no doubt."                     
[7] "That's not the judgment we need."               
[8] "That promise is our greatest inheritance."      

print("George W Bush")

[1] "George W Bush"

speech.df=tbl_df(sentence.list)%>%
  filter(File=="GeorgeWBush", type=="nomin", Term==1, word.count>=4)%>%
  select(sentences, anger:trust)
speech.df=as.data.frame(speech.df)
as.character(speech.df$sentences[apply(speech.df[,-1], 2, which.max)])

[1] "On the other side of that wall are poverty and prison, addiction and despair."                                           
[2] "We're proud of you."                                                                                                     
[3] "But they've got it backwards."                                                                                           
[4] "And at the earliest possible date, my administration will deploy missile defenses to guard against attack and blackmail."
[5] "I appreciate his friendship."                                                                                            
[6] "On the other side of that wall are poverty and prison, addiction and despair."                                           
[7] "They had their chance."                                                                                                  
[8] "Corporations are responsible to treat their workers fairly and to leave the air and waters clean."                       

print("Donald Trump")

[1] "Donald Trump"

speech.df=tbl_df(sentence.list)%>%
  filter(File=="DonaldJTrump", type=="nomin", Term==1, word.count>=5)%>%
  select(sentences, anger:trust)
speech.df=as.data.frame(speech.df)
as.character(speech.df$sentences[apply(speech.df[,-1], 2, which.max)])

[1] "Once again, France is the victim of brutal Islamic terrorism."                                                   
[2] "God bless you, and good night!"                                                                                  
[3] "I have visited the laid-off factory workers, and the communities crushed by our horrible and unfair trade deals."
[4] "Once again, France is the victim of brutal Islamic terrorism."                                                   
[5] "God bless you, and good night!"                                                                                  
[6] "Three were killed, and three were very very badly injured."                                                      
[7] "God bless you, and good night!"                                                                                  
[8] "God bless you, and good night!"                                                                                  

Clustering of emotions

heatmap.2(cor(sentence.list%>%filter(type=="inaug")%>%select(anger:trust)), 
          scale = "none", 
          col = bluered(100), , margin=c(6, 6), key=F,
          trace = "none", density.info = "none")
par(mar=c(4, 6, 2, 1))

emo.means=colMeans(select(sentence.list, anger:trust)>0.01)
col.use=c("red2", "darkgoldenrod1", 
            "chartreuse3", "blueviolet",
            "darkgoldenrod2", "dodgerblue3", 
            "darkgoldenrod1", "darkgoldenrod1")
barplot(emo.means[order(emo.means)], las=2, col=col.use[order(emo.means)], horiz=T, main="Inaugural Speeches")

presid.summary=tbl_df(sentence.list)%>%
  filter(type=="nomin", File%in%sel.comparison)%>%
  #group_by(paste0(type, File))%>%
  group_by(File)%>%
  summarise(
    anger=mean(anger),
    anticipation=mean(anticipation),
    disgust=mean(disgust),
    fear=mean(fear),
    joy=mean(joy),
    sadness=mean(sadness),
    surprise=mean(surprise),
    trust=mean(trust)
    #negative=mean(negative),
    #positive=mean(positive)
  )
presid.summary=as.data.frame(presid.summary)
rownames(presid.summary)=as.character((presid.summary[,1]))
km.res=kmeans(presid.summary[,-1], iter.max=200,
              5)
fviz_cluster(km.res, 
             stand=F, repel= TRUE,
             data = presid.summary[,-1], xlab="", xaxt="n",
             show.clust.cent=FALSE)

Step 5: Data analysis — Topic modeling

For topic modeling, we prepare a corpus of sentence snipets as follows. For each speech, we start with sentences and prepare a snipet with a given sentence with the flanking sentences.

corpus.list=sentence.list[2:(nrow(sentence.list)-1), ]
sentence.pre=sentence.list$sentences[1:(nrow(sentence.list)-2)]
sentence.post=sentence.list$sentences[3:(nrow(sentence.list)-1)]
corpus.list$snipets=paste(sentence.pre, corpus.list$sentences, sentence.post, sep=" ")
rm.rows=(1:nrow(corpus.list))[corpus.list$sent.id==1]
rm.rows=c(rm.rows, rm.rows-1)
corpus.list=corpus.list[-rm.rows, ]

Text mining

docs <- Corpus(VectorSource(corpus.list$snipets))
writeLines(as.character(docs[[sample(1:nrow(corpus.list), 1)]]))

We propose to rebuild the key tools of missile defense, starting with the Navy cruisers that are the foundation of our missile defense capabilities in Europe, Asia, and the Middle East. The Obama-Clinton administration tried repeatedly to remove our cruisers from service, then refused to modernize these aging ships. We will start by modernizing our cruisers to provide the Ballistic Missile Defense capability our nation needs;

Text basic processing

Adapted from https://eight2late.wordpress.com/2015/09/29/a-gentle-introduction-to-topic-modeling-using-r/.

#remove potentially problematic symbols
docs <-tm_map(docs,content_transformer(tolower))
writeLines(as.character(docs[[sample(1:nrow(corpus.list), 1)]]))

now, folks, you cheered—and i thank you— but the government can only do so much. the private sector has to provide most of these jobs. so i want to say again, tonight i challenge every business person in america who has ever complained about the failure of the welfare system to try to hire somebody off welfare and try hard.

#remove punctuation
docs <- tm_map(docs, removePunctuation)
writeLines(as.character(docs[[sample(1:nrow(corpus.list), 1)]]))

and by putting the federal department of labor at the disposal of the workingman when in search of work we have effected the emancipation of the children of the country by releasing them from hurtful labor we have instituted a system of national aid in the building of highroads such as the country has been feeling after for a century

#Strip digits
docs <- tm_map(docs, removeNumbers)
writeLines(as.character(docs[[sample(1:nrow(corpus.list), 1)]]))

the people must once more unite and overcome the advocates of repudiation and must not relax their energy until the battle for public honor and honest money shall again triumph a congress which will sustain and if need be strengthen the present law can prevent a financial catastrophe which every lover of the republic is interested to avert not satisfied with assaulting the currency and credit of the government our political adversaries condemn the tariff enacted at the extra session of congress in  known as the dingley act passed in obedience to the will of the people expressed at the election in the preceding november a law which at once stimulated our industries opened the factories and mines and gave to the laborer and to the farmers fair returns for their toil and investment

#remove stopwords
docs <- tm_map(docs, removeWords, stopwords("english"))
writeLines(as.character(docs[[sample(1:nrow(corpus.list), 1)]]))

  transaction   foreign affairs   endeavored  cultivate  friendship   nations  especially         important relations   done  justice   occasions favored  favor  lawful  cherished mutual interests  intercourse  fair  equal terms   firmly convinced   act   conviction   nations   individuals  interests soundly calculated will ever  found inseparable   moral duties  history bears witness   fact   just nation  trusted   word  recourse    armaments  wars  bridle others

#remove whitespace
docs <- tm_map(docs, stripWhitespace)
writeLines(as.character(docs[[sample(1:nrow(corpus.list), 1)]]))

 well known blessings enjoyed fullest extent add peculiar satisfaction example capital punishment inflicted anyone crime high treason might admit competency government beneficent duties might doubt trials put test strength efficiency member great community nations

#Stem document
docs <- tm_map(docs,stemDocument)
writeLines(as.character(docs[[sample(1:nrow(corpus.list), 1)]]))

now turn issu overrid import elect generat cometh progress made build new structur peac world peac import partisanship five presid polit lifetimefranklin d

Topic modeling

Gengerate document-term matrices.

dtm <- DocumentTermMatrix(docs)
#convert rownames to filenames#convert rownames to filenames
rownames(dtm) <- paste(corpus.list$type, corpus.list$File,
                       corpus.list$Term, corpus.list$sent.id, sep="_")
rowTotals <- apply(dtm , 1, sum) #Find the sum of words in each Document

Reached total allocation of 3792Mb: see help(memory.size)Reached total allocation of 3792Mb: see help(memory.size)Reached total allocation of 3792Mb: see help(memory.size)Reached total allocation of 3792Mb: see help(memory.size)Reached total allocation of 3792Mb: see help(memory.size)Reached total allocation of 3792Mb: see help(memory.size)Reached total allocation of 3792Mb: see help(memory.size)Reached total allocation of 3792Mb: see help(memory.size)Reached total allocation of 3792Mb: see help(memory.size)Reached total allocation of 3792Mb: see help(memory.size)Reached total allocation of 3792Mb: see help(memory.size)Reached total allocation of 3792Mb: see help(memory.size)Reached total allocation of 3792Mb: see help(memory.size)Reached total allocation of 3792Mb: see help(memory.size)Reached total allocation of 3792Mb: see help(memory.size)Reached total allocation of 3792Mb: see help(memory.size)Reached total allocation of 3792Mb: see help(memory.size)Reached total allocation of 3792Mb: see help(memory.size)Reached total allocation of 3792Mb: see help(memory.size)Reached total allocation of 3792Mb: see help(memory.size)

dtm  <- dtm[rowTotals> 0, ]
corpus.list=corpus.list[rowTotals>0, ]

Run LDA

#Set parameters for Gibbs sampling
burnin <- 4000
iter <- 2000
thin <- 500
seed <-list(2003,5,63,100001,765)
nstart <- 5
best <- TRUE
#Number of topics
k <- 15
#Run LDA using Gibbs sampling
ldaOut <-LDA(dtm, k, method="Gibbs", control=list(nstart=nstart, 
                                                 seed = seed, best=best,
                                                 burnin = burnin, iter = iter, 
                                                 thin=thin))

Reached total allocation of 3792Mb: see help(memory.size)Reached total allocation of 3792Mb: see help(memory.size)Error in get(helpTopicsName, pos = rsEnvPos) : 
  object '.completions.helpTopics' not found

help(memory.size)
#write out results
#docs to topics
ldaOut.topics <- as.matrix(topics(ldaOut))
table(c(1:k, ldaOut.topics))

   1    2    3    4    5    6    7    8    9   10   11   12   13   14   15 
1507 1580 2486  953 1287  717  979 1649 1746 1492 1300 1688 1281 1182 1133 

write.csv(ldaOut.topics,file=paste("../out/LDAGibbs",k,"DocsToTopics.csv"))
#top 6 terms in each topic
ldaOut.terms <- as.matrix(terms(ldaOut,20))
write.csv(ldaOut.terms,file=paste("../out/LDAGibbs",k,"TopicsToTerms.csv"))
#probabilities associated with each topic assignment
topicProbabilities <- as.data.frame(ldaOut@gamma)
write.csv(topicProbabilities,file=paste("../out/LDAGibbs",k,"TopicProbabilities.csv"))

terms.beta=ldaOut@beta
terms.beta=scale(terms.beta)
topics.terms=NULL
for(i in 1:k){
  topics.terms=rbind(topics.terms, ldaOut@terms[order(terms.beta[i,], decreasing = TRUE)[1:7]])
}
topics.terms

      [,1]            [,2]       [,3]         [,4]         [,5]       [,6]       
 [1,] "fellowcitizen" "discharg" "enlighten"  "bestow"     "exert"    "zeal"     
 [2,] "aspir"         "surrend"  "decenc"     "concept"    "invent"   "yearn"    
 [3,] "dont"          "immigr"   "didnt"      "polic"      "russia"   "someon"   
 [4,] "deed"          "fist"     "dissatisfi" "leak"       "matthew"  "pessimist"
 [5,] "time"          "sinc"     "three"      "almost"     "took"     "six"      
 [6,] "lake"          "awak"     "evenhand"   "mar"        "rancor"   "dimens"   
 [7,] "understood"    "frustrat" "optimist"   "enthusiasm" "grim"     "nobil"    
 [8,] "war"           "forc"     "defens"     "europ"      "east"     "soviet"   
 [9,] "convent"       "campaign" "john"       "governor"   "reagan"   "speech"   
[10,] "currenc"       "commerci" "expans"     "cent"       "export"   "european" 
[11,] "bridg"         "unpreced" "edg"        "pioneer"    "beacon"   "bread"    
[12,] "job"           "educ"     "billion"    "technolog"  "elder"    "senior"   
[13,] "voic"          "journey"  "brave"      "gift"       "daughter" "boy"      
[14,] "sovereignti"   "tend"     "tendenc"    "sphere"     "confin"   "jealous"  
[15,] "law"           "amend"    "appoint"    "judg"       "disput"   "railroad" 
      [,7]        
 [1,] "emin"      
 [2,] "wrought"   
 [3,] "shes"      
 [4,] "americaan" 
 [5,] "ceremoni"  
 [6,] "secreci"   
 [7,] "fantasi"   
 [8,] "negoti"    
 [9,] "eisenhow"  
[10,] "government"
[11,] "noblest"   
[12,] "bigger"    
[13,] "bright"    
[14,] "usurp"     
[15,] "interst"   

ldaOut.terms

      Topic 1    Topic 2     Topic 3  Topic 4      Topic 5     Topic 6   Topic 7  
 [1,] "may"      "right"     "want"   "will"       "year"      "countri" "can"    
 [2,] "shall"    "nation"    "know"   "must"       "time"      "one"     "peopl"  
 [3,] "public"   "freedom"   "just"   "everi"      "now"       "great"   "make"   
 [4,] "duti"     "peopl"     "like"   "never"      "first"     "peopl"   "believ" 
 [5,] "upon"     "free"      "think"  "nation"     "made"      "mani"    "good"   
 [6,] "confid"   "human"     "back"   "respons"    "last"      "citizen" "chang"  
 [7,] "trust"    "equal"     "dont"   "make"       "histori"   "much"    "way"    
 [8,] "servic"   "liberti"   "get"    "now"        "come"      "part"    "thing"  
 [9,] "spirit"   "justic"    "that"   "continu"    "ago"       "place"   "need"   
[10,] "honor"    "civil"     "tell"   "lead"       "long"      "differ"  "problem"
[11,] "best"     "life"      "ive"    "whether"    "past"      "anoth"   "done"   
[12,] "patriot"  "democraci" "fight"  "need"       "two"       "polit"   "valu"   
[13,] "feel"     "progress"  "say"    "take"       "sinc"      "serv"    "find"   
[14,] "high"     "among"     "immigr" "support"    "still"     "person"  "life"   
[15,] "present"  "seek"      "someth" "give"       "end"       "fact"    "work"   
[16,] "experi"   "individu"  "talk"   "effort"     "administr" "ever"    "learn"  
[17,] "purpos"   "men"       "put"    "understand" "four"      "import"  "choic"  
[18,] "oblig"    "hold"      "even"   "task"       "record"    "call"    "truth"  
[19,] "principl" "full"      "happen" "requir"     "today"     "other"   "help"   
[20,] "interest" "republ"    "weve"   "bring"      "day"       "alway"   "well"   
      Topic 8    Topic 9      Topic 10   Topic 11   Topic 12   Topic 13 Topic 14   
 [1,] "nation"   "presid"     "busi"     "america"  "work"     "hope"   "govern"   
 [2,] "peac"     "parti"      "system"   "american" "famili"   "see"    "state"    
 [3,] "world"    "republican" "industri" "new"      "job"      "men"    "power"    
 [4,] "war"      "say"        "upon"     "let"      "tax"      "man"    "unit"     
 [5,] "forc"     "democrat"   "product"  "world"    "children" "day"    "union"    
 [6,] "polici"   "ask"        "increas"  "futur"    "million"  "love"   "constitut"
 [7,] "defens"   "elect"      "labor"    "live"     "help"     "stand"  "interest" 
 [8,] "strength" "thank"      "trade"    "better"   "care"     "god"    "principl" 
 [9,] "danger"   "tonight"    "public"   "promis"   "economi"  "live"   "general"  
[10,] "militari" "american"   "develop"  "togeth"   "pay"      "women"  "form"     
[11,] "use"      "friend"     "polici"   "build"    "school"   "young"  "limit"    
[12,] "arm"      "accept"     "use"      "opportun" "program"  "heart"  "within"   
[13,] "also"     "offic"      "tariff"   "home"     "educ"     "know"   "exist"    
[14,] "unit"     "convent"    "import"   "today"    "plan"     "land"   "upon"     
[15,] "strong"   "proud"      "protect"  "generat"  "creat"    "faith"  "action"   
[16,] "intern"   "said"       "revenu"   "centuri"  "home"     "across" "feder"    
[17,] "clear"    "leader"     "market"   "old"      "save"     "rememb" "institut" 
[18,] "maintain" "member"     "employ"   "challeng" "health"   "call"   "whole"    
[19,] "secur"    "hous"       "price"    "come"     "reduc"    "side"   "citizen"  
[20,] "europ"    "unit"       "farmer"   "begin"    "worker"   "fear"   "local"    
      Topic 15   
 [1,] "law"      
 [2,] "congress" 
 [3,] "question" 
 [4,] "shall"    
 [5,] "without"  
 [6,] "administr"
 [7,] "secur"    
 [8,] "execut"   
 [9,] "legisl"   
[10,] "constitut"
[11,] "made"     
[12,] "protect"  
[13,] "subject"  
[14,] "order"    
[15,] "enforc"   
[16,] "effect"   
[17,] "upon"     
[18,] "may"      
[19,] "act"      
[20,] "issu"     

Based on the most popular terms and the most salient terms for each topic, we assign a hashtag to each topic. This part require manual setup as the topics are likely to change.

topics.hash=c("Economy", "America", "Defense", "Belief", "Election", "Patriotism", "Unity", "Government", "Reform", "Temporal", "WorkingFamilies", "Freedom", "Equality", "Misc", "Legislation")
corpus.list$ldatopic=as.vector(ldaOut.topics)
corpus.list$ldahash=topics.hash[ldaOut.topics]
colnames(topicProbabilities)=topics.hash
corpus.list.df=cbind(corpus.list, topicProbabilities)

Clustering of topics

par(mar=c(1,1,1,1))
topic.summary=tbl_df(corpus.list.df)%>%
              filter(type%in%c("nomin", "inaug"), File%in%sel.comparison)%>%
              select(File, Economy:Legislation)%>%
              group_by(File)%>%
              summarise_each(funs(mean))
topic.summary=as.data.frame(topic.summary)
rownames(topic.summary)=topic.summary[,1]
# [1] "Economy"         "America"         "Defense"         "Belief"         
# [5] "Election"        "Patriotism"      "Unity"           "Government"     
# [9] "Reform"          "Temporal"        "WorkingFamilies" "Freedom"        
# [13] "Equality"        "Misc"            "Legislation"       
topic.plot=c(1, 13, 9, 11, 8, 3, 7)
print(topics.hash[topic.plot])

[1] "Economy"         "Equality"        "Reform"          "WorkingFamilies"
[5] "Government"      "Defense"         "Unity"          

heatmap.2(as.matrix(topic.summary[,topic.plot+1]), 
          scale = "column", key=F, 
          col = bluered(100),
          cexRow = 0.9, cexCol = 0.9, margins = c(8, 8),
          trace = "none", density.info = "none")

# [1] "Economy"         "America"         "Defense"         "Belief"         
# [5] "Election"        "Patriotism"      "Unity"           "Government"     
# [9] "Reform"          "Temporal"        "WorkingFamilies" "Freedom"        
# [13] "Equality"        "Misc"            "Legislation"       
 
par(mfrow=c(5, 1), mar=c(1,1,2,0), bty="n", xaxt="n", yaxt="n")
topic.plot=c(1, 13, 14, 15, 8, 9, 12)
print(topics.hash[topic.plot])

[1] "Economy"     "Equality"    "Misc"        "Legislation" "Government"  "Reform"     
[7] "Freedom"    

speech.df=tbl_df(corpus.list.df)%>%filter(File=="GeorgeBush", type=="nomin",Term==1)%>%select(sent.id, Economy:Legislation)
speech.df=as.matrix(speech.df)
speech.df[,-1]=replace(speech.df[,-1], speech.df[,-1]<1/15, 0.001)
speech.df[,-1]=f.smooth.topic(x=speech.df[,1], y=speech.df[,-1])
plot.stacked(speech.df[,1], speech.df[,topic.plot+1], 
             xlab="Sentences", ylab="Topic share", main="George Bush, Nomination")

[1] 0.05345318 0.10690636 0.16035954 0.21381272 0.26726590 0.32071908 0.37417225

speech.df=tbl_df(corpus.list.df)%>%filter(File=="WilliamJClinton", type=="nomin", Term==1)%>%select(sent.id, Economy:Legislation)
speech.df=as.matrix(speech.df)
speech.df[,-1]=replace(speech.df[,-1], speech.df[,-1]<1/15, 0.001)
speech.df[,-1]=f.smooth.topic(x=speech.df[,1], y=speech.df[,-1])
plot.stacked(speech.df[,1], speech.df[,topic.plot+1],
            xlab="Sentences", ylab="Topic share", main="Bill Clinton, Nomination")

[1] 0.05477343 0.10954687 0.16432030 0.21909373 0.27386716 0.32864060 0.38341403

speech.df=tbl_df(corpus.list.df)%>%filter(File=="GeorgeWBush", type=="nomin", Term==1)%>%select(sent.id, Economy:Legislation)
speech.df=as.matrix(speech.df)
speech.df[,-1]=replace(speech.df[,-1], speech.df[,-1]<1/15, 0.001)
speech.df[,-1]=f.smooth.topic(x=speech.df[,1], y=speech.df[,-1])
plot.stacked(speech.df[,1], speech.df[,topic.plot+1], 
            xlab="Sentences", ylab="Topic share", main="George W Bush, Nomination")

[1] 0.05298909 0.10597818 0.15896728 0.21195637 0.26494546 0.31793455 0.37092364

speech.df=tbl_df(corpus.list.df)%>%filter(File=="BarackObama", type=="nomin", Term==1)%>%select(sent.id, Economy:Legislation)
speech.df=as.matrix(speech.df)
speech.df[,-1]=replace(speech.df[,-1], speech.df[,-1]<1/15, 0.001)
speech.df[,-1]=f.smooth.topic(x=speech.df[,1], y=speech.df[,-1])
plot.stacked(speech.df[,1], speech.df[,topic.plot+1],
            xlab="Sentences", ylab="Topic share", main="Barack Obama, Nomination")

[1] 0.05220546 0.10441092 0.15661638 0.20882184 0.26102731 0.31323277 0.36543823

speech.df=tbl_df(corpus.list.df)%>%filter(File=="DonaldJTrump", type=="nomin")%>%select(sent.id, Economy:Legislation)
speech.df=as.matrix(speech.df)
speech.df[,-1]=replace(speech.df[,-1], speech.df[,-1]<1/15, 0.001)
speech.df[,-1]=f.smooth.topic(x=speech.df[,1], y=speech.df[,-1])
plot.stacked(speech.df[,1], speech.df[,topic.plot+1],
            xlab="Sentences", ylab="Topic share", main="Donald Trump, Nomination")

[1] 0.04597271 0.09194541 0.13791812 0.18389083 0.22986353 0.27583624 0.32180895

# [1] "Economy"         "America"         "Defense"         "Belief"         
# [5] "Election"        "Patriotism"      "Unity"           "Government"     
# [9] "Reform"          "Temporal"        "WorkingFamilies" "Freedom"        
# [13] "Equality"        "Misc"            "Legislation"       
par(mfrow=c(5, 1), mar=c(1,1,2,0), bty="n", xaxt="n", yaxt="n")
topic.plot=c(1, 13, 14, 15, 8, 9, 12)
print(topics.hash[topic.plot])

[1] "Economy"     "Equality"    "Misc"        "Legislation" "Government"  "Reform"     
[7] "Freedom"    

speech.df=tbl_df(corpus.list.df)%>%filter(File=="GeorgeBush", type=="inaug", Term==1)%>%select(sent.id, Economy:Legislation)
speech.df=as.matrix(speech.df)
speech.df[,-1]=replace(speech.df[,-1], speech.df[,-1]<1/15, 0.001)
speech.df[,-1]=f.smooth.topic(x=speech.df[,1], y=speech.df[,-1])
plot.stacked(speech.df[,1], speech.df[,topic.plot+1],
             xlab="Sentences", ylab="Topic share", main="George Bush, inaugural Speeches")

[1] 0.05515922 0.11031843 0.16547765 0.22063687 0.27579609 0.33095530 0.38611452

speech.df=tbl_df(corpus.list.df)%>%filter(File=="WilliamJClinton", type=="inaug", Term==1)%>%select(sent.id, Economy:Legislation)
speech.df=as.matrix(speech.df)
speech.df[,-1]=replace(speech.df[,-1], speech.df[,-1]<1/15, 0.001)
speech.df[,-1]=f.smooth.topic(x=speech.df[,1], y=speech.df[,-1])
plot.stacked(speech.df[,1], speech.df[,topic.plot+1],
             xlab="Sentences", ylab="Topic share", main="William J Clinton, inaugural Speeches")

[1] 0.04966348 0.09932696 0.14899044 0.19865392 0.24831740 0.29798088 0.34764436

speech.df=tbl_df(corpus.list.df)%>%filter(File=="GeorgeWBush", type=="inaug", Term==1)%>%select(sent.id, Economy:Legislation)
speech.df=as.matrix(speech.df)
speech.df[,-1]=replace(speech.df[,-1], speech.df[,-1]<1/15, 0.001)
speech.df[,-1]=f.smooth.topic(x=speech.df[,1], y=speech.df[,-1])
plot.stacked(speech.df[,1], speech.df[,topic.plot+1],
             xlab="Sentences", ylab="Topic share", main="George W. Bush, inaugural Speeches")

[1] 0.05479929 0.10959859 0.16439788 0.21919718 0.27399647 0.32879576 0.38359506

speech.df=tbl_df(corpus.list.df)%>%filter(File=="BarackObama", type=="inaug", Term==1)%>%select(sent.id, Economy:Legislation)
speech.df=as.matrix(speech.df)
speech.df[,-1]=replace(speech.df[,-1], speech.df[,-1]<1/15, 0.001)
speech.df[,-1]=f.smooth.topic(x=speech.df[,1], y=speech.df[,-1])
plot.stacked(speech.df[,1], speech.df[,topic.plot+1],
             xlab="Sentences", ylab="Topic share", main="Barack Obama, inaugural Speeches")

[1] 0.04853656 0.09707311 0.14560967 0.19414622 0.24268278 0.29121933 0.33975589

# [1] "Economy"         "America"         "Defense"         "Belief"         
# [5] "Election"        "Patriotism"      "Unity"           "Government"     
# [9] "Reform"          "Temporal"        "WorkingFamilies" "Freedom"        
# [13] "Equality"        "Misc"            "Legislation"       
par(mfrow=c(5, 1))
topic.plot=c(1, 13, 14, 15, 8, 9, 12)
print(topics.hash[topic.plot])

[1] "Economy"     "Equality"    "Misc"        "Legislation" "Government"  "Reform"     
[7] "Freedom"    

speech.df=tbl_df(corpus.list.df)%>%filter(File=="RonaldReagan", type=="farewell")%>%select(sent.id, Economy:Legislation)
speech.df=as.matrix(speech.df)
speech.df[,-1]=replace(speech.df[,-1], speech.df[,-1]<1/15, 0.001)
speech.df[,-1]=f.smooth.topic(x=speech.df[,1], y=speech.df[,-1])
plot.stacked(speech.df[,1], speech.df[,topic.plot+1],
             xlab="Sentences", ylab="Topic share", main="Ronald Reagan, Farewell Speeches")

[1] 0.04998233 0.09996466 0.14994699 0.19992932 0.24991166 0.29989399 0.34987632

speech.df=tbl_df(corpus.list.df)%>%filter(File=="GeorgeBush", type=="farewell")%>%select(sent.id, Economy:Legislation)
speech.df=as.matrix(speech.df)
speech.df[,-1]=replace(speech.df[,-1], speech.df[,-1]<1/15, 0.001)
speech.df[,-1]=f.smooth.topic(x=speech.df[,1], y=speech.df[,-1])
plot.stacked(speech.df[,1], speech.df[,topic.plot+1],
             xlab="Sentences", ylab="Topic share", main="George Bush, Farewell Speeches")

[1] 0.05151229 0.10302458 0.15453687 0.20604916 0.25756145 0.30907374 0.36058603

speech.df=tbl_df(corpus.list.df)%>%filter(File=="WilliamJClinton", type=="farewell")%>%select(sent.id, Economy:Legislation)

Warning: Unknown column 'sentences'
Warning: Unknown column 'sentences'
Warning: Unknown column 'sentences'
Warning: Unknown column 'sentences'
Warning: Unknown column 'sentences'
Warning: Unknown column 'sentences'
Warning: Unknown column 'sentences'
Warning: Unknown column 'sentences'
Warning: Unknown column 'sentences'
Warning: Unknown column 'sentences'

speech.df=as.matrix(speech.df)
speech.df[,-1]=replace(speech.df[,-1], speech.df[,-1]<1/15, 0.001)
speech.df[,-1]=f.smooth.topic(x=speech.df[,1], y=speech.df[,-1])
plot.stacked(speech.df[,1], speech.df[,topic.plot+1],
             xlab="Sentences", ylab="Topic share", main="William J. Clinton, Farewell Speeches")

[1] 0.04670749 0.09341499 0.14012248 0.18682997 0.23353746 0.28024496 0.32695245

speech.df=tbl_df(corpus.list.df)%>%filter(File=="GeorgeWBush", type=="farewell")%>%select(sent.id, Economy:Legislation)
speech.df=as.matrix(speech.df)
speech.df[,-1]=replace(speech.df[,-1], speech.df[,-1]<1/15, 0.001)
speech.df[,-1]=f.smooth.topic(x=speech.df[,1], y=speech.df[,-1])
plot.stacked(speech.df[,1], speech.df[,topic.plot+1],
             xlab="Sentences", ylab="Topic share", main="George W Bush, Farewell Speeches")

[1] 0.05571697 0.11143394 0.16715091 0.22286788 0.27858484 0.33430181 0.39001878

speech.df=tbl_df(corpus.list.df)%>%filter(File=="BarackObama", type=="farewell")%>%select(sent.id, Economy:Legislation)
speech.df=as.matrix(speech.df)
speech.df[,-1]=replace(speech.df[,-1], speech.df[,-1]<1/15, 0.001)
speech.df[,-1]=f.smooth.topic(x=speech.df[,1], y=speech.df[,-1])
plot.stacked(speech.df[,1], speech.df[,topic.plot+1],
             xlab="Sentences", ylab="Topic share", main="Barack Obama, Farewell Speeches")

[1] 0.04720441 0.09440882 0.14161323 0.18881764 0.23602205 0.28322647 0.33043088

speech.df=tbl_df(corpus.list.df)%>%filter(type=="nomin", word.count<20)%>%select(sentences, Economy:Legislation)
as.character(speech.df$sentences[apply(as.data.frame(speech.df[,-1]), 2, which.max)])

 [1] "With great respect, I am Very truly yours, J."                                                             
 [2] "It has been wrought through party government."                                                             
 [3] "But not a single hunter in Arkansas or New Hampshire or Illinois or anyplace else missed a hunting season."
 [4] "It will be decided by you."                                                                                
 [5] "Well, I should—pardon me—I didn't put a period in there."                                                  
 [6] "They share the same piece of land, but they are different from one another."                               
 [7] "My friends, as President, I'm going to be setting goals for our country;"                                  
 [8] "We have helped to bring democracy to Haiti and peace to Bosnia."                                           
 [9] "Mr."                                                                                                       
[10] "Of great importance and still in a highly undeveloped state are our transportation routes by waterways."   
[11] "Let us resolve to build that bridge."                                                                      
[12] "My tax cuts will not undermine our economy, they will speed economic growth."                              
[13] "Crowds spontaneously began singing America the Beautiful or The Battle Hymn of the Republic."              
[14] "Its enforcement requires, therefore, independent but coincident action of both agencies."                  
[15] "With respect to notice, the Democratic platform contains no recommendation."                               

names(speech.df)[-1]

 [1] "Economy"         "America"         "Defense"         "Belief"         
 [5] "Election"        "Patriotism"      "Unity"           "Government"     
 [9] "Reform"          "Temporal"        "WorkingFamilies" "Freedom"        
[13] "Equality"        "Misc"            "Legislation"    

presid.summary=tbl_df(corpus.list.df)%>%
  filter(type=="inaug", File%in%sel.comparison)%>%
  select(File, Economy:Legislation)%>%
  group_by(File)%>%
  summarise_each(funs(mean))
presid.summary=as.data.frame(presid.summary)
rownames(presid.summary)=as.character((presid.summary[,1]))
km.res=kmeans(scale(presid.summary[,-1]), iter.max=200,
              5)
fviz_cluster(km.res, 
             stand=T, repel= TRUE,
             data = presid.summary[,-1],
             show.clust.cent=FALSE)